{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "92ab305f-ac76-43e6-a849-c57f7356d4cb",
   "metadata": {},
   "source": [
    "# GPT3 Complex Prompt on Strategy QA with 'Let's think step by step'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "3ba60447-e65b-4ea5-9da4-bb7df7fbf49c",
   "metadata": {},
   "outputs": [],
   "source": [
    "import openai\n",
    "import re\n",
    "import time\n",
    "import json\n",
    "\n",
    "import numpy as np\n",
    "\n",
    "from tqdm import tqdm\n",
    "from pprint import pprint\n",
    "from tenacity import retry, stop_after_attempt, wait_chain, wait_fixed"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "38419ccb-05c6-4dce-a657-e8150c411e44",
   "metadata": {},
   "outputs": [],
   "source": [
    "openai.api_key = \"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "a8535d96-fd53-4568-a988-86809bd36f64",
   "metadata": {},
   "outputs": [],
   "source": [
    "@retry(wait=wait_chain(*[wait_fixed(3) for i in range(3)] +\n",
    "                       [wait_fixed(5) for i in range(2)] +\n",
    "                       [wait_fixed(10)]))\n",
    "def completion_with_backoff(**kwargs):\n",
    "    return openai.Completion.create(**kwargs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "158636d2-a9c4-4d53-813a-f51b0a79e169",
   "metadata": {},
   "outputs": [],
   "source": [
    "train_data = json.load(open('data/strategyqa_train.json'))\n",
    "train_filtered = json.load(open('data/strategyqa_train_filtered.json'))\n",
    "train_data_paragraphs = json.load(open('data/strategyqa_train_paragraphs.json'))\n",
    "test_data = json.load(open('data/strategyqa_test.json'))\n",
    "dev_idx = np.load('data/dev_idx.npy')\n",
    "dev_data = [train_data[i] for i in dev_idx]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "b3c60318-fecb-4425-847b-381cc162bcd4",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "490"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(test_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bad1a332-3ec9-4c05-b8d6-993d96d8388a",
   "metadata": {},
   "source": [
    "# Original Prompt + Step, Acc 66.87"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "3919ba03-e9d3-47e2-97a5-a7de0161bb1a",
   "metadata": {},
   "outputs": [],
   "source": [
    "prompt_original_step = open('lib_prompt/prompt_original_step.txt').read()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "fa226522-54c1-4206-93a8-38259f449f4a",
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Q: Do hamsters provide food for any animals?\n",
      "A: Let's think step by step\n",
      "Hamsters are prey animals. \n",
      "Prey are food for predators. \n",
      "Thus, hamsters provide food for some animals. \n",
      "So the answer is yes.\n",
      "\n",
      "Q: Could Brooke Shields succeed at University of Pennsylvania?\n",
      "A: Let's think step by step\n",
      "Brooke Shields went to Princeton University. \n",
      "Princeton University is about as academically rigorous as the University of Pennsylvania. \n",
      "Thus, Brooke Shields could also succeed at the University of Pennsylvania. \n",
      "So the answer is yes.\n",
      "\n",
      "Q: Yes or no: Hydrogen’s atomic number squared exceeds number of Spice Girls?\n",
      "A: Let's think step by step\n",
      "Hydrogen has an atomic number of 1. 1 squared is 1. \n",
      "There are 5 Spice Girls. \n",
      "Thus, Hydrogen’s atomic number squared is less than 5. \n",
      "So the answer is no.\n",
      "\n",
      "Q: Yes or no: Is it common to see frost during some college commencements?\n",
      "A: Let's think step by step\n",
      "College commencement ceremonies can happen in December, May, and June. \n",
      "December is in the winter, so there can be frost. \n",
      "Thus, there could be frost at some commencements. \n",
      "So the answer is yes.\n",
      "\n",
      "Q: Yes or no: Could a llama birth twice during War in Vietnam (1945-46)?\n",
      "A: Let's think step by step\n",
      "The War in Vietnam was 6 months. \n",
      "The gestation period for a llama is 11 months, which is more than 6 months. \n",
      "Thus, a llama could not give birth twice during the War in Vietnam. \n",
      "So the answer is no. \n",
      "\n",
      "Q: Yes or no: Would a pear sink in water?\n",
      "A: Let's think step by step\n",
      "The density of a pear is about 0.6g/cm3, which is less than water. \n",
      "Objects less dense than water float. \n",
      "Thus, a pear would float. \n",
      "So the answer is no.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(prompt_original_step)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "fcb173f2-7b75-4084-bbf5-7f182a1a1823",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 800/800 [33:22<00:00,  2.50s/it]\n"
     ]
    }
   ],
   "source": [
    "acc = 0\n",
    "total = 0\n",
    "with open('outputs/dev_gpt3_original_step.txt', 'w') as fd:\n",
    "    for d in tqdm(dev_data):\n",
    "        q = d['question']\n",
    "        a = d['answer']\n",
    "        if(a is True): a = 'yes'\n",
    "        else: a = 'no'\n",
    "        \n",
    "        prompt_q = prompt_original_step + '\\nQ: ' + q + '\\n'\n",
    "        prompt_q += 'A:'\n",
    "\n",
    "        response = completion_with_backoff(model=\"text-davinci-002\", \n",
    "                                            prompt=prompt_q, \n",
    "                                            temperature=0, \n",
    "                                            max_tokens=256)\n",
    "\n",
    "        ans_model = response['choices'][0]['text']\n",
    "        fd.write('Q: %s\\nA_model:\\n%s\\nA:\\n%s\\n\\n' % (q, ans_model, a))\n",
    "        \n",
    "        ans_ = ans_model.split('answer is ')\n",
    "        if(len(ans_) > 1 and 'yes' in ans_[1]): ans_ = 'yes'\n",
    "        else: ans_ = 'no'\n",
    "        \n",
    "        if(ans_ == a): acc += 1\n",
    "        total += 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "f495a79e-7b67-4276-a8c1-5b50d4a67e55",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total 800 correct 535 acc 0.6687\n"
     ]
    }
   ],
   "source": [
    "print('Total %d correct %d acc %.4f' % (total, acc, acc / total))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3861e803-e35b-43cd-bad5-162818f2c33b",
   "metadata": {},
   "source": [
    "# Complex Prompt + Step, Acc 73.00"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "ba447cde-33a2-4bef-a42b-e758efb5e57b",
   "metadata": {},
   "outputs": [],
   "source": [
    "prompt_complex_step = open('lib_prompt/prompt_complex_step.txt').read()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "c1480a39-9160-4b8b-8bfa-d4fc32bcc69a",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 800/800 [49:30<00:00,  3.71s/it]\n"
     ]
    }
   ],
   "source": [
    "acc = 0\n",
    "total = 0\n",
    "with open('outputs/dev_gpt3_complex_step.txt', 'w') as fd:\n",
    "    for d in tqdm(dev_data):\n",
    "        q = d['question']\n",
    "        a = d['answer']\n",
    "        if(a is True): a = 'yes'\n",
    "        else: a = 'no'\n",
    "        \n",
    "        prompt_q = prompt_complex_step + '\\nQ: ' + q + '\\n'\n",
    "        prompt_q += 'A:'\n",
    "\n",
    "        response = completion_with_backoff(model=\"text-davinci-002\", \n",
    "                                            prompt=prompt_q, \n",
    "                                            temperature=0, \n",
    "                                            max_tokens=256)\n",
    "\n",
    "        ans_model = response['choices'][0]['text']\n",
    "        fd.write('Q: %s\\nA_model:\\n%s\\nA:\\n%s\\n\\n' % (q, ans_model, a))\n",
    "        \n",
    "        ans_ = ans_model.split('answer is ')\n",
    "        if(len(ans_) > 1 and 'yes' in ans_[1]): ans_ = 'yes'\n",
    "        else: ans_ = 'no'\n",
    "        \n",
    "        if(ans_ == a): acc += 1\n",
    "        total += 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "298cedb8-9d35-4476-8ed2-a7730ff092fd",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total 800 correct 584 acc 0.7300\n"
     ]
    }
   ],
   "source": [
    "print('Total %d correct %d acc %.4f' % (total, acc, acc / total))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "04444ddc-7dcc-48ab-83c6-e48a38ab9135",
   "metadata": {},
   "source": [
    "# Simple Prompt + Step, Acc 71.13"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "48c30272-1f32-42d9-a305-7ddb1196261d",
   "metadata": {},
   "outputs": [],
   "source": [
    "prompt_simple_step = open('lib_prompt/prompt_simple_step.txt').read()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "10a065a4-025d-492f-8ed0-74eb138dbd2f",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 800/800 [36:14<00:00,  2.72s/it]\n"
     ]
    }
   ],
   "source": [
    "acc = 0\n",
    "total = 0\n",
    "with open('outputs/dev_gpt3_simple_step.txt', 'w') as fd:\n",
    "    for d in tqdm(dev_data):\n",
    "        q = d['question']\n",
    "        a = d['answer']\n",
    "        if(a is True): a = 'yes'\n",
    "        else: a = 'no'\n",
    "        \n",
    "        prompt_q = prompt_simple_step + '\\nQ: ' + q + '\\n'\n",
    "        prompt_q += 'A:'\n",
    "\n",
    "        response = completion_with_backoff(model=\"text-davinci-002\", \n",
    "                                            prompt=prompt_q, \n",
    "                                            temperature=0, \n",
    "                                            max_tokens=256)\n",
    "\n",
    "        ans_model = response['choices'][0]['text']\n",
    "        fd.write('Q: %s\\nA_model:\\n%s\\nA:\\n%s\\n\\n' % (q, ans_model, a))\n",
    "        \n",
    "        ans_ = ans_model.split('answer is ')\n",
    "        if(len(ans_) > 1 and 'yes' in ans_[1]): ans_ = 'yes'\n",
    "        else: ans_ = 'no'\n",
    "        \n",
    "        if(ans_ == a): acc += 1\n",
    "        total += 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "eb221e2a-d114-4534-8412-0d27c655ea4a",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total 800 correct 569 acc 0.7113\n"
     ]
    }
   ],
   "source": [
    "print('Total %d correct %d acc %.4f' % (total, acc, acc / total))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "32fced41-0af1-4197-954c-36fb9e4fa820",
   "metadata": {},
   "source": [
    "# Complex Prompt + Step + Conjunction, Acc 77.00"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "b97c6339-eb5e-4bbe-95a8-d3e522cccda0",
   "metadata": {},
   "outputs": [],
   "source": [
    "prompt_complex_conjunction = open('lib_prompt/prompt_complex_conjunction.txt').read()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "64b6e3fb-06cb-4b24-a117-528574949d6f",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 800/800 [54:22<00:00,  4.08s/it]\n"
     ]
    }
   ],
   "source": [
    "acc = 0\n",
    "total = 0\n",
    "with open('outputs/dev_gpt3_complex_conjunction.txt', 'w') as fd:\n",
    "    for d in tqdm(dev_data):\n",
    "        q = d['question']\n",
    "        a = d['answer']\n",
    "        if(a is True): a = 'yes'\n",
    "        else: a = 'no'\n",
    "        \n",
    "        prompt_q = prompt_complex_conjunction + '\\nQ: ' + q + '\\n'\n",
    "        prompt_q += 'A:'\n",
    "\n",
    "        response = completion_with_backoff(model=\"text-davinci-002\", \n",
    "                                            prompt=prompt_q, \n",
    "                                            temperature=0, \n",
    "                                            max_tokens=256)\n",
    "\n",
    "        ans_model = response['choices'][0]['text']\n",
    "        fd.write('Q: %s\\nA_model:\\n%s\\nA:\\n%s\\n\\n' % (q, ans_model, a))\n",
    "        \n",
    "        ans_ = ans_model.split('answer is ')\n",
    "        if(len(ans_) > 1 and 'yes' in ans_[1]): ans_ = 'yes'\n",
    "        else: ans_ = 'no'\n",
    "        \n",
    "        if(ans_ == a): acc += 1\n",
    "        total += 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "59574604-6775-4607-8807-1be74dcb9340",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total 800 correct 616 acc 0.7700\n"
     ]
    }
   ],
   "source": [
    "print('Total %d correct %d acc %.4f' % (total, acc, acc / total))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
